Structured light depth sensing

ABSTRACT

At least one waveguide is configured to receive coherent illumination light and direct the coherent illumination light to an object as a first light projection and second light projection. At least one interference pattern is generated by interference between the first light projection and the second light projection. A camera captures interference images of a plurality of phase-shifted interference images and a depth from an object may be determined from the phase-shifted interference images.

TECHNICAL FIELD

This disclosure relates generally to optics, and in particular to depthsensing.

BACKGROUND INFORMATION

A variety of techniques for range and depth sensing have been developed.For example, a stereo triangulation technique includes imaging an objectwith two different cameras and determining a distance to the objectusing corresponding points in the images from the two cameras. Time ofFlight (TOF) is another existing technique that may include transmittinglaser pulses and measuring the time of flight between the transmittedand reflected laser pulse to calculate the depth of an object thatreflected the laser pulse. However, these techniques are limited by thecontext they are deployed in. TOF techniques, for example, struggle invery near-range contexts because resolving the short time of flight ofthe pulses is difficult.

SUMMARY

Embodiments according to the invention are in particular disclosed inthe attached claims directed to a head mounted display (HMD), an opticalstructure, and a method, wherein any feature mentioned in one claimcategory, e.g. HMD, can be claimed in another claim category, e.g.optical structure, method, system, storage medium, and computer programproduct, as well. The dependencies or references back in the attachedclaims are chosen for formal reasons only. However, any subject matterresulting from a deliberate reference back to any previous claims (inparticular multiple dependencies) can be claimed as well, so that anycombination of claims and the features thereof is disclosed and can beclaimed regardless of the dependencies chosen in the attached claims.The subject-matter which can be claimed comprises not only thecombinations of features as set out in the attached claims but also anyother combination of features in the claims, wherein each featurementioned in the claims can be combined with any other feature orcombination of other features in the claims. Furthermore, any of theembodiments and features described or depicted herein can be claimed ina separate claim and/or in any combination with any embodiment orfeature described or depicted herein or with any of the features of theattached claims.

In an embodiment, a Head Mounted Display (HMD) may comprise:

at least one camera configured to image an eyebox area; and

an optical structure to be positioned in a view of a user of the HMD,wherein the optical structure includes a first, second, and thirdwaveguide configured to receive coherent illumination light and directthe coherent illumination light to the eyebox area as a first lightprojection and a second light projection to generate a first, second,and third interference pattern corresponding to the first, second, andthird waveguides, respectively, and wherein the camera is configured tocapture the first, second, and third interference patterns.

The first, second, and third waveguide each may include a firstextraction feature to direct the first light projection to a first areaand a second extraction feature to direct the second light projection toan area different from the first area, and an overlap of the first lightprojection and the second light projection from each respectivewaveguide may generate the respective interference pattern on the eyeboxarea.

In an embodiment, a HMD may comprise:

a first coherent light source configured to emit first coherentillumination light, wherein the first waveguide is configured to receivethe first coherent illumination light;

a second coherent light source configured to emit second coherentillumination light, wherein the second waveguide is configured toreceive the second coherent illumination light; and

a third coherent light source configured to emit third coherentillumination light, wherein the third waveguide is configured to receivethe third coherent illumination light, and wherein the first and secondextraction feature of the first, second, and third waveguides are spaceddifferently to generate a first, second, and third interference pattern,respectively, that are phase-shifted from each other, and furtherwherein the first, second, and third coherent illumination light are thesame wavelength.

The third interference pattern may be phase-shifted from the secondinterference pattern by 120 degrees, and the second interference patternmay be phase-shifted from the first interference pattern by 120 degrees.

In an embodiment, a HMD may comprise:

a fourth coherent light source, wherein the optical structure includes afourth waveguide configured to generate a fourth interference pattern inthe eyebox area, wherein the fourth interference patter is phase-shiftedfrom the third interference pattern by 90 degrees, the thirdinterference pattern is phase-shifted from the second interferencepattern by 90 degrees, and the second interference pattern isphase-shifted from the first interference pattern by 90 degrees.

In an embodiment, a HMD may comprise:

a first coherent light source configured to emit first coherentillumination light, wherein the first waveguide is configured to receivethe first coherent illumination light;

a second coherent light source configured to emit second coherentillumination light, wherein the second waveguide is configured toreceive the second coherent illumination light; and

a third coherent light source configured to emit third coherentillumination light, wherein the third waveguide is configured to receivethe third coherent illumination light, wherein the first, second, andthird coherent illumination light are at different infrared wavelengths.

At least one of the first extraction feature or the second extractionfeature may include a reflective coating configured to direct the firstlight projection or the second light projection.

The first, second, and third interference patterns may includesinusoidal fringe patterns.

The coherent illumination light may be coherent infrared light.

The camera may include a filter that passes an infrared wavelength rangematched to the coherent illumination light and rejects other wavelengthranges.

A coherent light source providing the coherent illumination light mayinclude at least one of an infrared vertical-cavity surface-emittinglaser (VCSEL), laser diode, superluminescent light emitting diode (SLED)with high spatial coherency, or distributed feedback laser (DFB).

An imaging axis of the camera in relation to a projection axis betweenthe first and second light projection may be greater than 30 degrees.

The first, second, and third waveguides may be configured as single-modewaveguides.

In an embodiment, an optical structure may comprise: a transparentlayer; and

a first, second, and third waveguide configured to guide coherentillumination light, wherein the first, second, and third waveguides arecoupled with the transparent layer, wherein each of the first, second,and third waveguide include:

a first extraction feature configured to direct the coherentillumination light as a first light projection in an eyebox area; and

a second extraction feature configured to direct the coherentillumination light as a second light projection in the eyebox area,wherein the first light projection and the second light projectioninterfere to generate an interference pattern.

The first and second extraction feature of the first, second, and thirdwaveguides may be spaced differently to generate a first, second, andthird interference pattern, respectively, that are phase-shifted fromeach other.

The coherent illumination light may be centered around a near-infraredwavelength.

At least one of the first extraction feature or the second extractionfeature may include a reflective coating configured to direct the firstlight projection or the second light projection, the reflective coatingmay include a dielectric or metallic coating configured to reflect thefirst light projection or the second light projection and pass visiblelight.

In an embodiment, a method of near-eye depth sensing may comprise:

generating a first, second, and third interference pattern in an eyeboxarea;

capturing, with a camera, a first, second, and third interference imagecorresponding to the first, second, and third interference pattern,respectively; and

generating at least one eye-depth value based at least in part on thefirst, second, and third interference image.

Generating the first, second, and third interference patterns mayinclude directing coherent illumination light into at least onewaveguide, and the at least one waveguide may include a first extractionfeature to direct a first light projection to a first area and a secondextraction feature to direct a second light projection to second areadifferent from the first area.

The first, second, and third interference patterns may be generatedsequentially and captured sequentially by the camera as the first,second, and third interference images.

In an embodiment according to the invention, one or morecomputer-readable non-transitory storage media may embody software thatis operable when executed to perform a method according to the inventionor any of the above-mentioned embodiments.

In an embodiment according to the invention, a system may comprise: oneor more processors; and at least one memory coupled to the processorsand comprising instructions executable by the processors, the processorsoperable when executing the instructions to perform a method accordingto the invention or any of the above-mentioned embodiments.

In an embodiment according to the invention, a computer program product,preferably comprising a computer-readable non-transitory storage media,may be operable when executed on a data processing system to perform amethod according to the invention or any of the above-mentionedembodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the invention aredescribed with reference to the following figures, wherein likereference numerals refer to like parts throughout the various viewsunless otherwise specified.

FIGS. 1A and 1B illustrate an example head mounted display that includesa camera and waveguides for generating phase-shifted interferencepatterns for near-eye depth sensing, in accordance with an embodiment ofthe disclosure.

FIGS. 2A and 2B illustrate an example waveguide having two extractionfeatures for generating interference patterns, in accordance with anembodiment of the disclosure.

FIG. 3 illustrates an example head mounted display directing lightprojections to form interference patterns, in accordance with anembodiment of the disclosure.

FIG. 4 illustrates an example optical substrate and three waveguidesgenerating three emission cones, in accordance with an embodiment of thedisclosure.

FIG. 5 illustrates example phase-shifted interference images, inaccordance with an embodiment of the disclosure.

FIG. 6 illustrates an example flow chart for a method of near-rangedepth sensing, in accordance with an embodiment of the disclosure.

FIG. 7 illustrates a top down view of a system for near-range depthsensing, in accordance with an embodiment of the disclosure.

FIG. 8 illustrates example equations that may be utilized for generatinga depth map of an object, in accordance with an embodiment of thedisclosure.

DETAILED DESCRIPTION

Embodiments of depth sensing systems, devices, and methods are describedherein. In the following description, numerous specific details are setforth to provide a thorough understanding of the embodiments. Oneskilled in the relevant art will recognize, however, that the techniquesdescribed herein can be practiced without one or more of the specificdetails, or with other methods, components, materials, etc. In otherinstances, well-known structures, materials, or operations are not shownor described in detail to avoid obscuring certain aspects.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, theappearances of the phrases “in one embodiment” or “in an embodiment” invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments.

Throughout this specification, several terms of art are used. Theseterms are to take on their ordinary meaning in the art from which theycome, unless specifically defined herein or the context of their usewould clearly suggest otherwise.

Embodiments of depth sensing systems, devices, and methods described inthe disclosure may facilitate high resolution near-range depth sensing.The near-range depth sensing embodiments may be specifically implementedin a head mounted display (HMD) for determining a three-dimensionaldepth mapping of an eye of a wearer of an HMD and the three-dimensionaldepth mapping may be used for eye-tracking or other purposes.

Prior depth sensing techniques include Time of Flight and patternedprojections. Time of Flight (ToF) techniques include transmitting laserpulses and measuring the time of flight between the transmitted andreflected laser pulse to calculate the depth of an object that reflectedthe laser pulse. However, ToF systems struggle in very near-rangecontexts because resolving the short time of flight of the pulses(propagating at the speed of light) is difficult. An example patternedprojection technique projects a pattern of dots onto an object and theposition and/or the size of the projected dots on the object are used todetermine a depth of the object. However, the resolution of patternprojection approaches is limited by the number of dots projected ontothe surface of the object.

In embodiments of the disclosure, phase-shifted interference patternsare generated by waveguides having first and second extraction features.The waveguides may receive coherent illumination light and provide thecoherent illumination light to the first and second extraction featuresthat are spaced a distance apart. The coherent illumination light may beprovided by an infrared laser light source, for example. The first andsecond extraction features direct the received coherent illuminationlight to an eyebox area as first and second light projections. Aninterference pattern is formed by the interference of the first andsecond light projections and captured by a camera. Additional waveguidesmay generate additional interference patterns that are phase-shiftedfrom the first interference patterns. For example, a minimum of threewaveguides with extraction features may generate three or moreinterference patterns on an eye (or other object) where the threeinterference patterns are phase-shifted by 120 degrees. Intensity valuesfrom the three phase-shifted interference patterns can then be utilizedto determine a depth map of the object.

Embodiments of the disclosure may be particularly useful in near-rangedepth sensing context where ToF systems struggle to generate accuratemappings. Additionally, embodiments of the disclosure may offer higherresolution depth-mappings than pattern projection techniques because theembodiments of the disclosure are not limited by a number of projecteddots. Furthermore, the disclosed waveguides and extraction features maybe so small as to be unnoticeable even when placed in view of a wearerof an HMD. Therefore, the embodiments of the disclosure may have theadded of advantage of being placed within a field of view of an eye of awearer of an HMD such that the user still has a clear view of incomingscene light. Embodiments of the disclosure may also provide a very lowpower depth sensing system because the disclosed coherent illuminationlight sources may only be mW devices that are not required to be oncontinuously. These and other embodiments are described in detail withrespect to FIGS. 1A-8 below.

FIG. 1A illustrates an example HMD 100 that includes a camera andwaveguides that project light to generate phase differing interferencepatterns for near-eye depth sensing, in accordance with an embodiment ofthe disclosure. HMD 100 includes frame 114 coupled to arms 111A and111B. Lenses 121A and 121B are mounted to frame 114. Lenses 121 may beprescription lenses matched to a particular wearer of HMD ornon-prescription lenses. The illustrated HMD 100 is configured to beworn on or about a head of a user of the HMD.

In FIG. 1A, each lens 121 may include a waveguide 150 to direct imagelight generated by a display 130 to an eyebox area for viewing by awearer of HMD 100. Display 130 may include an LCD, an organic lightemitting diode (OLED) display, micro-LED display, quantum dot display,pico-projector, or liquid crystal on silicon (LCOS) display fordirecting image light to a wearer of HMD 100.

The frame 114 and arms 111 of the HMD 100 may include supportinghardware of HMD 100. HMD 100 may include any of processing logic, wiredand/or wireless data interface for sending and receiving data, graphicprocessors, and one or more memories for storing data andcomputer-executable instructions. In one embodiment, HMD 100 may beconfigured to receive wired power. In one embodiment, HMD 100 isconfigured to be powered by one or more batteries. In one embodiment,HMD 100 may be configured to receive wired data including video data viaa wired communication channel. In one embodiment, HMD 100 is configuredto receive wireless data including video data via a wirelesscommunication channel.

Lenses 121 may appear transparent to a user to facilitate augmentedreality or mixed reality where a user can view scene light from theenvironment around her while also receiving image light directed to hereye(s) by waveguide(s) 150. Consequently, lenses 121 may be considered(or include) an optical combiner. In some embodiments, image light isonly directed into one eye of the wearer of HMD 100. In an embodiment,both displays 130A and 130B are included to direct image light intowaveguides 150A and 150B, respectively.

The example HMD 100 of FIG. 1A includes a plurality of waveguides 160coupled to receive coherent illumination light from coherentillumination source(s) 170 and direct the coherent illumination light toan eyebox area as first and second light projections. The first andsecond light projection generate an interference pattern on the eye of awearer of HMD 100. The coherent illumination light may be infrared lightand light source(s) 170 may be any of an infrared vertical-cavitysurface-emitting laser (VCSEL) laser diode, superluminescent lightemitting diode (SLED) with high spatial coherency, or distributedfeedback laser (DFB). In one embodiment, the infrared coherentillumination light is centered around 850 nm. Camera 147 is configuredto image the eyebox area and the eye of a wearer of HMD 100. Camera 147may be mounted on the inside of the temple of HMD 100 and image theeyebox area directly, as illustrated in FIG. 1A. Camera 147 may also bemounted elsewhere. In one embodiment, camera 147 is positioned toreceive an infrared image from the eyebox area reflected from a “hotmirror” combiner include in lens 121B. The “hot mirror” combiner isconfigured to reflect infrared light while passing visible scene lightto the eye of a wearer of HMD 100. Camera 147 may include a filter thatpasses an infrared wavelength range matched to the coherent illuminationlight of light source(s) 170 and rejects other wavelength ranges.Although camera 147, waveguides 160, and coherent illumination source(s)170 are illustrated on only one side of HMD 100, they of course may beduplicated on the other side of HMD 100 to facilitate near-eye depthsensing for both eyes of a wearer of HMD 100.

FIG. 1B illustrates a zoomed in view of a portion of example HMD 100, inaccordance with an embodiment of the disclosure. FIG. 1B illustrateswaveguide 160A, 160B, and 160C coupled to receive coherent illuminationlight from coherent illumination light sources 170A, 170B, and 170C,respectively. The coherent illumination light from a given coherentlight source may be both spatially coherent and temporally coherent.Coherent illumination light may be narrow-band infrared light emittedfrom a laser source.

FIG. 2A illustrates an example waveguide 260 that could be utilized aswaveguides 160A, 160B, and 160C, in accordance with an embodiment of thedisclosure. Waveguide 260 receive coherent illumination light 280 fromcoherent illumination source 270. Coherent illumination source 270 canbe coupled into the waveguide 260 using a prism, micro-lens, or gratingstructures. Coherent illumination light 280 propagates in waveguide 260from a first end of the waveguide (near coherent illumination source270) to a second end of the waveguide 260 that is opposite the firstend. Extraction features 261 and 262 receive the coherent illuminationlight 280.

First extraction feature 261 is configured to receive the coherentillumination light 280 and direct a first light projection 281 to afirst area that includes at least a portion of an eyebox area. The firstarea may cover the entire eyebox area in some embodiments. Secondextraction feature 262 is configured to receive the coherentillumination light 280 and direct a second light projection 282 to asecond area that overlaps the first area. The overlap between the firstlight projection 281 in the first area and the second light projection282 in the second area generates an interference pattern.

In one embodiment, first extraction feature 261 and second extractionfeature 262 are both biconical surfaces fabricated in parallel at thesecond end of waveguide 260. The orientation and the curvature of thetwo biconical surfaces can be tuned to vary the orientation of theemission cone and the divergence of the cone of light projection 281and/or 282 to generate the interference pattern on the proper positionin the eyebox area. A highly reflective coating (e.g. metal ordielectric coating) may overlay the biconical surfaces to maximize thelight directed toward the eyebox.

In embodiments where waveguide 260 includes a ridge waveguide, the ridgemay be written onto an optical substrate (e.g. lens 121) and may belaminated to the optical substrate with an index-matched material wherethe refractive index of the lamination/bonding material is matched tothe optical substrate. The refractive index of the optical substrate maybe lower than index of waveguide 260 so that the total internalreflectivity (TIR) of waveguide 260 may be maintained to confinecoherent illumination light 280 to waveguide 260. Waveguide 260 mayfacilitate single-mode functionality where illumination light 280propagates through waveguide 260 in the transverse mode and propagatesparallel to the length of waveguide 260 (from the first end to thesecond end). In an embodiment, the length of waveguide 260 isapproximately 20 mm. Waveguides 170A, 170B, and 170C may have differinglengths, in some embodiments.

First extraction feature 261 is spaced a distance from second extractionfeature 262 so the first light projection 281 interferes with the secondlight projection 282 to form a structured light interference pattern onthe eye. First extraction feature 261 and second extraction feature 262may simulate a point source and emit conical shaped light projection 281and conical shaped light projection 282.

FIG. 2B illustrates a cross-section view of example waveguide 260 inFIG. 2A along lines A-A′, in accordance with an embodiment of thedisclosure. The cross section of waveguide 260 may be rectangular. Thecross section of waveguide 260 may be square. In one embodiment,dimension 211 is approximately 10 microns and dimension 212 isapproximately 10 microns. In one embodiment, dimension 211 isapproximately 8 microns and dimension 212 is approximately 8 microns.Waveguide 260 may have dimensions small enough to be unnoticeable by awearer of HMD 100.

FIG. 3 illustrates an example HMD 300 that includes waveguidesprojecting interference patterns in an eyebox area, in accordance withan embodiment of the disclosure. In FIG. 3, waveguide(s) 160 areconfigured to direct first light projection 281 and second lightprojection 282 to the eyebox area to form an interference patterngenerated by the interference of light 281 and 282.

FIG. 4 illustrates an optical substrate 421 and three waveguides 460emitting three emission cones to generate three interference patternsthat are phase-shifted from one another, in accordance with anembodiment of the disclosure. In FIG. 4, coherent illumination sources470A, 470B, and 470C are optically coupled to waveguides 460A, 460B, and460C, respectively, to provide coherent illumination light (e.g. light280) to each waveguide. Each waveguide 460 includes two extractionfeatures such as extraction features 261/262 to direct first and secondprojections 481/482 to the eyebox area. The first emission cone 483Acorresponding to waveguide 460A includes first light projection 481A andsecond light projection 482A. The second emission cone 483Bcorresponding to waveguide 460B includes first light projection 481B andsecond light projection 482B and the third emission cone 483Ccorresponding to waveguide 460C includes first light projection 481C andsecond light projection 482C. In some embodiments, only one emissioncone 483 is incident on the eyebox area at a given point in time as theemission cones 483 may be time-multiplexed with each other.

FIG. 5 illustrates images of phase-shifted interference patternsgenerated by different waveguides, in accordance with an embodiment ofthe disclosure. FIG. 5 includes images 583A, 583B, and 583C of theinterference patterns on an eye 502 generated by emission cones 483A,483B, and 483C, respectively. In one embodiment, the interferencepatterns generated by emission cones 483A/B/C are phase-shifted by 120degrees. For example, image 583B may be an image of an interferencepattern that is shifted 120 degrees from the interference pattern ofimage 583A and image 583C may be an image of an interference patternthat is shifted 240 degrees from the interference pattern of image 583A.Since extraction features 261/262 of a waveguide 260/460 generate afirst light projection and a second light projection that is highlycoherent, the interference pattern includes a stable periodic fringe onthe eyebox area. Notably, the dark and light fringes in images 583A/B/Care positioned differently—consistent with the phase-shifted differencebetween the image 583A/B/C.

Returning again to FIG. 4, the extraction features of waveguides 460 maybe separated by different distances in order to generate interferencespatterns that are phase-shifted from each other. For example, theextraction features (e.g. 261/262) of waveguide 460A may be spaced afirst distance from each other, the extraction features of waveguide460B may be spaced a second distance from each other, and the extractionfeatures of waveguide 460C may be spaced a third distance from eachother. In one embodiment, the spacing between the waveguides themselves(e.g. 460A/B/C) determines the phase-shift of the interference patternsfrom one another. To generate interference images 583A/B/C, the coherentillumination source 470 may be activated for a first time period and acamera (e.g. 147) may capture image 583A of the interference patterngenerated by emission cone 483A during the first time period. After thefirst time period, coherent illumination source 470A may be deactivated(turned off) and coherent illumination source 470B may be activated fora second time period where the camera captures image 583B of theinterference pattern generated by emission cone 483B during the secondtime period. After the third time period, coherent illumination source470B may be deactivated (turned off) and coherent illumination source470C may be activated for a third time period where the camera capturesimage 583C of the interference pattern generated by emission cone 483Cduring the third time period. In other words, the coherent illuminationlight sources may be activated sequentially in order to generatephase-shifted interference patterns on the eye of a wearer of HMD andthe image capturing of the camera will be coordinated to capture thephase-shifted interference patterns. In this context, the coherentillumination sources may have the same central wavelength.

In contrast to the sequential time-multiplexed scenario described in theabove paragraph, a different embodiment allows for simultaneouslycapturing the three or more interference patterns generated by emissioncones 483. To facilitate this simultaneous capture of the three or moreinterference patterns, each waveguide is illuminated with a coherentillumination source having a different wavelength. For example, thefirst waveguide may be illuminated with a first wavelength, the secondwaveguide may be illuminated with a second wavelength, and the thirdwaveguide may be illuminated with a third wavelength. A high-resolutionimage sensor may be configured with filters matched to the first,second, and third wavelength. For example, instead of the conventionalRed/Green/Blue (RGB) filters applied to imaging pixels included insub-pixels of image sensors, the RGB filters are replaced with a firstfilter that passes the first wavelength and rejects other wavelengths, asecond filter that passes the second wavelength and rejects otherwavelengths, and a third filter that passes the third wavelength andrejects other wavelengths. Therefore, when the electronic shutter of theimage sensor captures an image, there are actually three sub imagesincluded in the images. The first sub-image is of the first interferencepattern having the first wavelength, the second sub-image is of thesecond interference pattern having the second wavelength, and the thirdsub-image is of the third interference pattern having the thirdwavelength.

In some embodiments, a single coherent illumination source is used tosequentially provide coherent illumination light to waveguides 460A/B/C.In this embodiment, an optical fiber may carry coherent illuminationlight from a single coherent light source to an optical switch that willbe sequentially switched to provide the coherent illumination light tothe waveguides 460A/B/C, one at a time.

In some embodiments, coherent illumination sources 460A/B/C may havedifferent wavelengths in order to generate phase-shifted interferencepatterns. In this embodiment, extraction features 261 and 262 ofwaveguides 460A/B/C may be spaced the same distance apart and thewavelength difference in the illumination sources is relied upon togenerate the phase-shifted interference patterns. The extractionfeatures 261 and 262 of waveguides 460A/B/C may also be spaceddifferently even when the wavelength difference in the illuminationsources is different.

In yet another embodiment, a swept laser provides the coherentillumination light to each waveguide 460A/B/C at different times. Thewavelength of a swept laser can be tuned. The wavelength of some sweptlasers is tuned according to a drive current of the laser, for example.Hence, a swept laser with its output sequentially switched to waveguides460A/B/C may sequentially provide coherent illumination light to thewaveguides 460A/B/C at different times. The swept laser may be driven sothat different wavelengths of coherent illumination light provided toeach waveguide 460A/B/C corresponds with a 120-degree phase shift on theinterference patterns generated by emission cones 483A/B/C.

While the implementation of embodiments of this disclosure have thus farreferences a phase shift of 120 degrees, those skilled in the artappreciate that depth information could also be determined withinterference patterns having different phase shifts such as fourinterference patterns having phases of 0 degrees, 90 degrees, 180degrees, and 270 degrees, for example. In this embodiment, fourwaveguides 460 may need to be provided to generate the four interferencepatterns (phase-shifted by 90 degrees from each other). Otherphase-shift values (e.g. 60 degrees) are also contemplated.

FIG. 6 illustrates an example process of near-range depth sensing inaccordance with an embodiment of the disclosure. The order in which someor all of the process blocks appear in process 600 should not be deemedlimiting. Rather, one of ordinary skill in the art having the benefit ofthe present disclosure will understand that some of the process blocksmay be executed in a variety of orders not illustrated, or even inparallel.

Near-range depth sensing may be defined as sensing a depth of an objectthat is within 30 cm. In one embodiment, near-range depth sensingincludes sensing a depth of an object (e.g. an eye) within 50 mm. Thedepth data may be used to determine eye gaze, capturing biometricfeatures of a user of an HMD for recognition purposes, and/or to analyzeeye-movements for different tasks.

In process block 605, a first, second, and third interference pattern isgenerated in an eyebox area. The first, second, and third interferencepatterns may be generated by emission cones 483A/B/C corresponding towaveguides 460A/B/C, for example. In one embodiment, generating thefirst, second, and third interference patterns includes directingcoherent illumination light into at least one waveguide where the atleast one waveguide includes a first extraction feature to direct thefirst light projection to a first area and a second extraction featureto direct a second light projection to second area different from thefirst area.

In process block 610, first, second, and third interference images (e.g.583A/B/C) corresponding with the first, second, and third interferencepatterns are captured. The first, second, and third interferencepatterns may be generated sequentially and captured sequentially by thecamera (e.g. camera 147).

In process block 615, at least one eye-depth value is generated based atleast in part on the first, second, and third interference image. Insome embodiments, a three-dimensional image of the eye is generatedusing the plurality of pixel values from a pixel array of the camera.

The eye of the wearer of an HMD will distort the fringes of thephase-shifted interference patterns projected onto the eye and after thethree images 583A/B/C of phase-shifted interference patterns arecaptured by the camera, three-dimensional depth information (indimension “z”) can be reconstructed using a triangulation model of phasedisparities using the pixel disparities in dimensions “x” and “y”corresponding with a pixel array of the camera. For example, the cameramay include a CMOS pixel array having “i” rows and “j” columns thatcorrespond with dimension x and y. Each phase disparity calculationassociated with each pixel yields an eye-depth value (in dimension z)for that pixel. Hence, the eye-depth value of each pixel combined withthe x and y dimensions of the pixel array allows for a three-dimensionaldepth mapping of the eye.

An eye-depth value or three-dimensional mapping of the eye may be usedfor eye-tracking purposes and a display image or display images directedto an eye of the user may be changed based on the eye-depth value orthree-dimensional mapping. In one embodiment, a position of asoftware-generated blur filter is applied to display images in responseto the eye-depth value(s) to provide focusing cues for a user of an HMD.

The example algorithm of generating eye-depth values orthree-dimensional mappings of the eye using phase-shifted interferencepatterns described in U.S. non-provisional patent application Ser. No.16/025,837 can be utilized in accordance with embodiments of thisdisclosure. U.S. non-provisional patent application Ser. No. 16/025,837filed Jul. 2, 2018 is hereby incorporated by reference.

Processing logic included in an HMD may be coupled to sequentiallyactivate coherent illumination sources 470 and also coupled to a camera(e.g. 147) to sequentially capture the phase-shifted interferencepatterns to execute example process 600 and other techniques describedherein.

FIG. 7 illustrates a top down view of a system 700 for near-field depthsensing, in accordance with an embodiment of the disclosure. System 700may be included in an HMD and be used for triangulation in near-eyedepth sensing, in some contexts. Although FIG. 7 illustrates only twoextraction features 761 and 762, it is understood that system 700 mayalso be used with additional extraction features corresponding withadditional waveguides. System 700 includes an object 702, a firstextraction feature 761, a second extraction feature 762, and an imagesensor 747. In FIG. 7, lens 743 is configured to focus light onto imagesensor 747 to provide imaging in a field of view 745 between dashedlines 745A and 745B. Lens 743 is disposed in plane 791 and imaging axis795 is normal to plane 791, in the illustrated embodiment. In someembodiments, lens 743 is offset from plane 791 while still beingpositioned at the center of image sensor 747. Lens 743 is spaced fromimage sensor 747 by a focal length f. Imaging axis 795 may run through acentral optical axis of lens 743 and through a middle of a pixel arrayhaving x rows and y columns. Axis 795 is spaced from axis 793 byseparation diameter D in the x direction.

Extraction features 761 and 762 are spaced a distance “a” apart. Firstextraction feature 761 directs coherent illumination light provided by awaveguide (e.g. 260, not illustrated) toward object 702 as first lightprojection 781 and second extraction feature 762 directs the coherentillumination light provided by the waveguide toward object 702 as secondlight projection 782. First light projection 781 and second lightprojection 782 may have conical emission cones. The overlap of light 781and 782 is illustrated as overlapping light 783 and the interference oflight 781 and 782 forms an interference pattern on object 702. Theinterference pattern projected on object 702 may be described by asinusoidally varying intensity pattern. Axis 792 runs through extractionfeatures 761 and 762. ϕ represents an angle between plane 791 and axis792. Axis 793 runs through a mid-point between extraction features 761and 762 while also being normal to plane 791. Projection axis 796 runsthrough the mid-point between extraction features 761 and 762 andrepresents a middle of overlapping light 783.

The intensity measured by pixel 749 of image sensor 747 corresponds tothe intensity on object surface 702 at position 709 within plane 797.Plane 797 may be within a distance z of plane 791. Distance z may beless than 50 mm, in some embodiments. The intensity of the interferencepattern light at position 709 becomes incident upon pixel 749propagating along optical path 799. Pixel 749 is offset from imagingaxis 795 by distance x₁. Axis 798 runs from the midpoint betweenextraction feature 761 and 762 to the middle of position 709. θ definesthe angle from axis 796 to axis 798 and a defines the angle between axis798 and axis 793. The angles θ and a may be different for differentpixels on image sensor 747 and can be determined, for example, through acamera calibration procedure. To integrate system 700 into an HMD suchas HMD 100, a calibrated baseline dimension D to form a triangulationbetween the illuminator (e.g. extraction features 761 and 762) and theimage sensor 747 may be taken.

To determine the depth z for a given position (e.g. 709) of objectsurface 702, the equations of FIG. 8 may be utilized, in accordance withan embodiments of the disclosure. Equation 801 provides the intensity ata position x, y for a first interference image (e.g. image 583A).Equation 802 provides the intensity at a position x, y for a secondinterference image (e.g. image 583B) and equation 803 provides theintensity at a position x, y for a third interference image (e.g. image583C). In equations 801, 802, and 803, Ø represents the phase-shift ofthe phase-shifted interference patterns on object 702 generated byemission cones 483A/B/C, for example. Hence, ΔØ may be 120 degrees. Asdescribed above, the spacing of extraction features 761 and 762 may bedifferent in different waveguides to provide the desired phase-shift Ø.Equation 804 provides for the wrapped phase calculation and equation 805provides the unwrapped phase calculation where N equals the number ofphase shifts used. Where ΔØ is 120 degrees, N may equal 3. If ΔØ was 90degrees, N may be 4, for example. Equation 806 provides the expectedintensity as a function of the angular position θ for a two-beaminterference pattern where λ is the wavelength of emitted light and a isthe spacing distance of the extraction features 761 and 762 emitting thelight. Equation 807 is an expanded version of equation 806 that includesan unwrapped phase component 808 including variable z where D is thebaseline between the camera and the illuminator, x₁ is the pixelposition on the camera focal array, and f is the effective focal lengthof the camera. Therefore, solving equation 807 for variable z providesthe depth z for a particular position 709 of the surface of an object702. And solving for a two-dimensional array of a plurality of positionsof object 702 provides a three-dimensional depth mapping of depth z ofthe surface of object 702.

Embodiments of the invention may include or be implemented inconjunction with an artificial reality system. Artificial reality is aform of reality that has been adjusted in some manner beforepresentation to a user, which may include, e.g., a virtual reality (VR),an augmented reality (AR), a mixed reality (MR), a hybrid reality, orsome combination and/or derivatives thereof. Artificial reality contentmay include completely generated content or generated content combinedwith captured (e.g., real-world) content. The artificial reality contentmay include video, audio, haptic feedback, or some combination thereof,and any of which may be presented in a single channel or in multiplechannels (such as stereo video that produces a three-dimensional effectto the viewer). Additionally, in some embodiments, artificial realitymay also be associated with applications, products, accessories,services, or some combination thereof, that are used to, e.g., createcontent in an artificial reality and/or are otherwise used in (e.g.,perform activities in) an artificial reality. The artificial realitysystem that provides the artificial reality content may be implementedon various platforms, including a head-mounted display (HMD) connectedto a host computer system, a standalone HMD, a mobile device orcomputing system, or any other hardware platform capable of providingartificial reality content to one or more viewers.

The term “processing logic” in this disclosure may include one or moreprocessors, microprocessors, multi-core processors, Application-specificintegrated circuits (ASIC), and/or Field Programmable Gate Arrays(FPGAs) to execute operations disclosed herein. In some embodiments,memories (not illustrated) are integrated into the processing logic tostore instructions to execute operations and/or store data. Processinglogic may also include analog or digital circuitry to perform theoperations in accordance with embodiments of the disclosure.

A “memory” or “memories” described in this disclosure may include one ormore volatile or non-volatile memory architectures. The “memory” or“memories” may be removable and non-removable media implemented in anymethod or technology for storage of information such ascomputer-readable instructions, data structures, program modules, orother data. Example memory technologies may include RAM, ROM, EEPROM,flash memory, CD-ROM, digital versatile disks (DVD), high-definitionmultimedia/data storage disks, or other optical storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other non-transmission medium that can be usedto store information for access by a computing device.

Communication channels may include or be routed through one or morewired or wireless communication utilizing IEEE 802.11 protocols,BlueTooth, SPI (Serial Peripheral Interface), I²C (Inter-IntegratedCircuit), USB (Universal Serial Port), CAN (Controller Area Network),cellular data protocols (e.g. 3G, 4G, LTE, 5G), optical communicationnetworks, Internet Service Providers (ISPs), a peer-to-peer network, aLocal Area Network (LAN), a Wide Area Network (WAN), a public network(e.g. “the Internet”), a private network, a satellite network, orotherwise.

A computing device may include a desktop computer, a laptop computer, atablet, a phablet, a smartphone, a feature phone, a server computer, orotherwise. A server computer may be located remotely in a data center orbe stored locally.

The processes explained above are described in terms of computersoftware and hardware. The techniques described may constitutemachine-executable instructions embodied within a tangible ornon-transitory machine (e.g., computer) readable storage medium, thatwhen executed by a machine will cause the machine to perform theoperations described. Additionally, the processes may be embodied withinhardware, such as an application specific integrated circuit (“ASIC”) orotherwise.

A tangible non-transitory machine-readable storage medium includes anymechanism that provides (i.e., stores) information in a form accessibleby a machine (e.g., a computer, network device, personal digitalassistant, manufacturing tool, any device with a set of one or moreprocessors, etc.). For example, a machine-readable storage mediumincludes recordable/non-recordable media (e.g., read only memory (ROM),random access memory (RAM), magnetic disk storage media, optical storagemedia, flash memory devices, etc.).

The above description of illustrated embodiments of the invention,including what is described in the Abstract, is not intended to beexhaustive or to limit the invention to the precise forms disclosed.While specific embodiments of, and examples for, the invention aredescribed herein for illustrative purposes, various modifications arepossible within the scope of the invention, as those skilled in therelevant art will recognize.

These modifications can be made to the invention in light of the abovedetailed description. The terms used in the following claims should notbe construed to limit the invention to the specific embodimentsdisclosed in the specification. Rather, the scope of the invention is tobe determined entirely by the following claims, which are to beconstrued in accordance with established doctrines of claiminterpretation.

What is claimed is:
 1. A Head Mounted Display (HMD) comprising: at leastone camera configured to image an eyebox area; and an optical structureto be positioned in a view of a user of the HMD, wherein the opticalstructure includes a first, a second, and a third waveguide, each of thefirst, second, and third waveguides configured to receive coherentillumination light and direct the coherent illumination light to theeyebox area as a first light projection and a second light projection togenerate a first, second, and third interference pattern correspondingto the first, second, and third waveguides, respectively, wherein thecoherent illumination light is coherent near-infrared light, and whereinthe camera is configured to capture the first, second, and thirdinterference patterns.
 2. The HMD of claim 1, wherein the first, second,and third waveguide each includes a first extraction feature to directthe first light projection a first area and a second extraction featureto direct the second light projection to an area different from thefirst area, and wherein an overlap of the first light projection and thesecond light projection from each respective waveguide generate therespective interference pattern on the eyebox area.
 3. The HMD of claim2 further comprising: a first coherent light source configured to emitfirst coherent illumination light, wherein the first waveguide isconfigured to receive the first coherent illumination light; a secondcoherent light source configured to emit second coherent illuminationlight, wherein the second waveguide is configured to receive the secondcoherent illumination light; and a third coherent light sourceconfigured to emit third coherent illumination light, wherein the thirdwaveguide is configured to receive the third coherent illuminationlight, and wherein the first and second extraction feature of the first,second, and third waveguides are spaced differently to generate a first,second, and third interference pattern, respectively, that arephase-shifted from each other, and further wherein the first, second,and third coherent illumination light are the same wavelength.
 4. TheHMD of claim 3, wherein the third interference pattern is phase-shiftedfrom the second interference pattern by 120 degrees, and wherein thesecond interference pattern is phase-shifted from the first interferencepattern by 120 degrees.
 5. The HMD of claim 3 further comprising: afourth coherent light source, wherein the optical structure includes afourth waveguide configured to generate a fourth interference pattern inthe eyebox area, wherein the fourth interference patter is phase-shiftedfrom the third interference pattern by 90 degrees, the thirdinterference pattern is phase-shifted from the second interferencepattern by 90 degrees, and the second interference pattern isphase-shifted from the first interference pattern by 90 degrees.
 6. TheHMD of claim 2 further comprising: a first coherent light sourceconfigured to emit first coherent illumination light, wherein the firstwaveguide is configured to receive the first coherent illuminationlight; a second coherent light source configured to emit second coherentillumination light, wherein the second waveguide is configured toreceive the second coherent illumination light; and a third coherentlight source configured to emit third coherent illumination light,wherein the third waveguide is configured to receive the third coherentillumination light, wherein the first, second, and third coherentillumination light are at different near-infrared wavelengths.
 7. TheHMD of claim 2, wherein at least one of the first extraction feature orthe second extraction feature includes a reflective coating configuredto direct the first light projection or the second light projection. 8.The HMD of claim 1, wherein the first, second, and third interferencepatterns includes sinusoidal fringe patterns.
 9. The HMD of claim 1,wherein the camera includes a filter that passes a near-infraredwavelength range matched to the coherent illumination light and rejectsother wavelength ranges.
 10. The HMD of claim 1, wherein a coherentlight source providing the coherent illumination light includes at leastone of near-infrared vertical-cavity surface-emitting laser (VCSEL),laser diode, superluminescent light emitting diode (SLED) with highspatial coherency, or distributed feedback laser (DFB).
 11. The HMD ofclaim 1, wherein an imaging axis of the camera in relation to aprojection axis between the first and second light projection is greaterthan 30 degrees.
 12. The HMD of claim 1, wherein the first, second, andthird waveguides are configured as single-mode waveguides.
 13. Anoptical structure comprising: a transparent layer; and a first, asecond, and a third waveguide, each configured to guide coherentillumination light, wherein the first, second, and third waveguides arecoupled with the transparent layer, wherein the coherent illuminationlight is coherent near-infrared light, and wherein each of the first,second, and third waveguide include: a first extraction featureconfigured to direct the coherent illumination light as a first lightprojection in an eyebox area; and a second extraction feature configuredto direct the coherent illumination light as a second light projectionin the eyebox area, wherein the first light projection and the secondlight projection interfere to generate an interference pattern.
 14. Theoptical structure of claim 13, wherein the first and second extractionfeature of the first, second, and third waveguides are spaceddifferently to generate a first, second, and third interference pattern,respectively, that are phase-shifted from each other.
 15. The opticalstructure of claim 13, wherein at least one of the first extractionfeature or the second extraction feature includes a reflective coatingconfigured to direct the first light projection or the second lightprojection, wherein the reflective coating includes a dielectric ormetallic coating configured to reflect the first light projection or thesecond light projection and pass visible light.
 16. A method of near-eyedepth sensing, the method comprising: generating a first lightprojection and a second light projection with each of a first waveguide,a second waveguide, and a third waveguide, wherein the first lightprojection and the second light projection are generated in response tocoherent near-infrared light directed by the first, second, and thirdwaveguides; generating a first, second, and third interference patternin an eyebox area in response to the first light projection and thesecond light projection generated by the first, second, and thirdwaveguides, respectively; capturing, with a camera, a first, second, andthird interference image corresponding to the first, second, and thirdinterference pattern, respectively; and generating at least oneeye-depth value based at least in part on the first, second, and thirdinterference image.
 17. The method of claim 16, wherein each of thefirst, second, and third waveguides include a first extraction featureto direct the first light projection to a first area and a secondextraction feature to direct the second light projection to second areadifferent from the first area.
 18. The method of claim 16, wherein thefirst, second, and third interference patterns are generatedsequentially and captured sequentially by the camera as the first,second, and third interference images.